In forecasting competitions, the traditional mechanism scores the predictions of each contestant against the outcome of each event, and the contestant with the highest total score wins. While it is well-known that this traditional mechanism can suffer from incentive issues, it is folklore that contestants will still be roughly truthful as the number of events grows. Yet thus far the literature lacks a formal analysis of this traditional mechanism. This paper gives the first such analysis. We first demonstrate that the ''long-run truthfulness'' folklore is false: even for arbitrary numbers of events, the best forecaster can have an incentive to hedge, reporting more moderate beliefs to increase their win probability. On the positive side, however, we show that two contestants will be approximately truthful when they have sufficient uncertainty over the relative quality of their opponent and the outcomes of the events, a case which may arise in practice.

In the field of machine learning and artificial intelligence, time series forecasting plays a pivotal role across various domains such as finance, healthcare, and weather. However, the task of selecting the most suitable forecasting method for a given dataset is a complex task due to the diversity of data patterns and characteristics. This research aims to address this challenge by proposing a comprehensive benchmark for evaluating and ranking time series forecasting methods across a wide range of datasets. This study investigates the comparative performance of many methods from two prominent time series forecasting frameworks, AutoGluon-Timeseries, and sktime to shed light on their applicability in different real-world scenarios. This research contributes to the field of time series forecasting by providing a robust benchmarking methodology, and facilitating informed decision-making when choosing forecasting methods for achieving optimal prediction.

This paper proposes a framework for developing forecasting models by streamlining the connections between core components of the developmental process. The proposed framework enables swift and robust integration of new datasets, experimentation on different algorithms, and selection of the best models. We start with the datasets of different issues and apply pre-processing steps to clean and engineer meaningful representations of time-series data. To identify robust training configurations, we introduce a novel mechanism of multiple cross-validation strategies. We apply different evaluation metrics to find the best-suited models for varying applications. One of the referent applications is our participation in the intelligent forecasting competition held by the United States Agency of International Development (USAID). Finally, we leverage the flexibility of the framework by applying different evaluation metrics to assess the performance of the models in inventory management settings.

Accuracy competition: This competition hosted by Walmart aimed to forecast daily sales of 3,049 products in 10 stores over a period of 28 days. Participants were required to forecast the sales of each product for each day of the competition using historical sales data provided by Walmart. This competition taught participants how to deal with a large dataset with multiple features and how to use various time-series forecasting techniques, such as ARIMA and Prophet. The Rossmann Store Sales competition: This competition aimed to forecast the daily sales of 1,115 Rossmann stores located in Germany. Participants were required to forecast sales for the next six weeks, taking into account factors such as promotions, school holidays, and store closures.

In this post, we explain how to use nixtlats and ydata-synthetic, open-source and free python libraries that allow you to generate synthetic data to train state-of-the-art deep learning models without any significant loss of data quality. We develop a deep learning forecasting pipeline without direct access to the original data and show that synthetic data has a minimal impact on the performance of the models. In the last decade, neural network-based forecasting methods have become ubiquitous in large-scale forecasting applications, transcending industry boundaries into academia, as it has redefined the state-of-the-art in many practical tasks like demand planning, electricity load forecasting, reverse logistics, weather forecasting, as well as forecasting competitions like the M4 and M5. However, one of the problems for those interested in creating forecasts is model development or software testing without using original data; this may be because the actual data takes time to collect, there are restrictions on its use, or the data simply does not exist. In many applications, the user does not want the model to have access to the actual data, in particular, if the model training is done in the cloud or outside one's infrastructure.

Earlier this year, my research group commissioned 6 questions for professional forecasters to predict about AI. They have financial incentives to produce accurate forecasts; the rewards total \$5k per question (\$30k total) and payoffs are (close to) a proper scoring rule, meaning forecasters are rewarded for outputting calibrated probabilities. You're in luck, because I'm going to answer each of these in the following sections! Feel free to skim to the ones that interest you the most. The particular questions were designed by my students Alex Wei, Collin Burns, Jean-Stanislas Denain, and Dan Hendrycks.

Usual statistical models apply a set of known relationships to a dataset. For example, exponential smoothing will have its way of estimating the underlying demand level and trend. On the other hand, machine learning is about letting an algorithm understand a dataset and its underlying relationships on its own. A Machine Learning algorithm will run through a dataset, look at data features, and (try to) pick up any underlying relationship. Choosing the correct data to feed to your model is tremendously important.

Many businesses and industries nowadays rely on large quantities of time series data making time series forecasting an important research area. Global forecasting models that are trained across sets of time series have shown a huge potential in providing accurate forecasts compared with the traditional univariate forecasting models that work on isolated series. However, there are currently no comprehensive time series archives for forecasting that contain datasets of time series from similar sources available for the research community to evaluate the performance of new global forecasting algorithms over a wide variety of datasets. In this paper, we present such a comprehensive time series forecasting archive containing 20 publicly available time series datasets from varied domains, with different characteristics in terms of frequency, series lengths, and inclusion of missing values. We also characterise the datasets, and identify similarities and differences among them, by conducting a feature analysis. Furthermore, we present the performance of a set of standard baseline forecasting methods over all datasets across eight error metrics, for the benefit of researchers using the archive to benchmark their forecasting algorithms.

With large quantities of data typically available nowadays, forecasting models that are trained across sets of time series, known as Global Forecasting Models (GFM), are regularly outperforming traditional univariate forecasting models that work on isolated series. As GFMs usually share the same set of parameters across all time series, they often have the problem of not being localised enough to a particular series, especially in situations where datasets are heterogeneous. We study how ensembling techniques can be used with generic GFMs and univariate models to solve this issue. Our work systematises and compares relevant current approaches, namely clustering series and training separate submodels per cluster, the so-called ensemble of specialists approach, and building heterogeneous ensembles of global and local models. We fill some gaps in the approaches and generalise them to different underlying GFM model types. We then propose a new methodology of clustered ensembles where we train multiple GFMs on different clusters of series, obtained by changing the number of clusters and cluster seeds. Using Feed-forward Neural Networks, Recurrent Neural Networks, and Pooled Regression models as the underlying GFMs, in our evaluation on six publicly available datasets, the proposed models are able to achieve significantly higher accuracy than baseline GFM models and univariate forecasting methods.

Competitions play an invaluable role in the field of forecasting, as exemplified through the recent M4 competition. The competition received attention from both academics and practitioners and sparked discussions around the representativeness of the data for business forecasting. Several competitions featuring real-life business forecasting tasks on the Kaggle platform has, however, been largely ignored by the academic community. We believe the learnings from these competitions have much to offer to the forecasting community and provide a review of the results from six Kaggle competitions. We find that most of the Kaggle datasets are characterized by higher intermittence and entropy than the M-competitions and that global ensemble models tend to outperform local single models. Furthermore, we find the strong performance of gradient boosted decision trees, increasing success of neural networks for forecasting, and a variety of techniques for adapting machine learning models to the forecasting task.